BDMS-470: Migrate Stratigraphy by jirhiker · Pull Request #393 · DataIntegrationGroup/OcotilloAPI

jirhiker · 2026-01-16T02:06:11Z

Why

This PR addresses the following problem / context:

Staging lacks a first-class legacy stratigraphy table, so Stratigraphy.csv rows aren’t preserved outside the association transfer.
Without a dedicated transfer + tests, stratigraphy CSV ingestion can regress silently.
Alembic/test/transfer flows don’t share a single schema reset path for stratigraphy legacy data.

How

Implementation summary - the following was changed / added / removed:

Added Alembic revision 1d2c3b4a5e67_create_nma_stratigraphy_table.py plus the Stratigraphy model (NMA_Stratigraphy table) and wired Thing’s relationship so legacy intervals can be queried off wells.
Built transfers/stratigraphy_legacy.py with a batch upsert pipeline, hooked it into the main transfer worker (guarded by TRANSFER_NMA_STRATIGRAPHY), and exposed metrics for the new table.
Created tests/test_stratigraphy_legacy.py to cover the model FK linkage and the transfer’s CSV ingestion path end to end.

Notes

Any special considerations, workarounds, or follow-up work to note?

Run the new transfer after wells load so PointID→Thing resolutions succeed.
The Alembic revision must be applied before running transfers/tests to ensure NMA_Stratigraphy exists.

chatgpt-codex-connector · 2026-01-16T02:06:20Z

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

Copilot

Pull request overview

This PR adds support for ingesting legacy stratigraphy data from Stratigraphy.csv into a dedicated NMA_Stratigraphy table. Previously, stratigraphy rows weren't preserved as first-class records outside of association transfers.

Changes:

Added NMA_Stratigraphy table model with relationship to Thing
Implemented batch upsert transfer pipeline for Stratigraphy.csv with environment flag control
Added tests covering model relationships and end-to-end CSV ingestion

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
alembic/versions/1d2c3b4a5e67_create_nma_stratigraphy_table.py	Creates NMA_Stratigraphy table schema with indexes and foreign key to Thing
db/nma_legacy.py	Defines Stratigraphy model with columns for lithology log data
db/thing.py	Adds stratigraphy_logs relationship to Thing model
transfers/stratigraphy_legacy.py	Implements StratigraphyLegacyTransferer with CSV parsing and batch upsert logic
transfers/transfer.py	Integrates stratigraphy legacy transfer into parallel and sequential execution paths
transfers/metrics.py	Adds nma_stratigraphy_metrics method for tracking transfer results
tests/test_stratigraphy_legacy.py	Tests model relationships and transfer functionality

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Copilot · 2026-01-16T04:54:46Z

+        if isinstance(value, str):
+            trimmed = value.strip()
+            return trimmed or None
+        return str(value)


The _string_value method converts non-string, non-None values to strings, which may silently coerce unexpected types. Consider adding validation or logging when converting non-string values to help catch data quality issues in the CSV.

Suggested change

return str(value)

# Log when coercing non-string values to string to help catch data-quality issues.

logger.warning(

"Coercing non-string value to string in StratigraphyLegacyTransferer._string_value: "

"value=%r (type=%s)",

value,

type(value).__name__,

)

try:

coerced = str(value)

except Exception:

logger.exception(

"Failed to coerce value to string in StratigraphyLegacyTransferer._string_value: "

"value=%r (type=%s)",

value,

type(value).__name__,

)

return None

trimmed = coerced.strip()

return trimmed or None

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

…or wells

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…nto BDMS-470-Migrate-Stratigraphy

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.

…nto BDMS-470-Migrate-Stratigraphy

Copilot

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.

feat: add NMA_Stratigraphy table and transfer functionality

7a4ca8a

jirhiker requested review from chasetmartin, Copilot, kbighorse, kblake and ksmuczynski January 16, 2026 02:06

jirhiker changed the title ~~feat: add NMA_Stratigraphy table and transfer functionality~~ BDMS-470: Migrate Stratigraphy Jan 16, 2026

Copilot AI reviewed Jan 16, 2026

View reviewed changes

Comment thread alembic/versions/1d2c3b4a5e67_create_nma_stratigraphy_table.py Outdated

Comment thread transfers/stratigraphy_legacy.py Outdated

Update alembic/versions/1d2c3b4a5e67_create_nma_stratigraphy_table.py

e834aad

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings January 16, 2026 04:52

Copilot AI reviewed Jan 16, 2026

View reviewed changes

Update transfers/stratigraphy_legacy.py

47bb8af

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings January 16, 2026 04:53

Copilot AI reviewed Jan 16, 2026

View reviewed changes

jirhiker added 2 commits January 15, 2026 22:00

fix: handle None data case in populate_fields method

06432fb

Merge branch 'staging' into BDMS-470-Migrate-Stratigraphy

aa19373

Copilot AI review requested due to automatic review settings January 16, 2026 05:04

Copilot AI reviewed Jan 16, 2026

View reviewed changes

Comment thread transfers/stratigraphy_legacy.py Outdated

jirhiker and others added 2 commits January 15, 2026 22:47

feat: enhance test setup with location and measuring point creation f…

adcdeb0

…or wells

Update transfers/stratigraphy_legacy.py

35b5d00

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings January 16, 2026 16:57

Copilot AI reviewed Jan 16, 2026

View reviewed changes

Comment thread tests/test_geospatial.py Outdated

jirhiker and others added 2 commits January 16, 2026 09:58

Merge branch 'staging' into BDMS-470-Migrate-Stratigraphy

8f2219d

Update tests/test_geospatial.py

fa53d47

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings January 16, 2026 16:59

jirhiker and others added 2 commits January 16, 2026 09:59

Merge remote-tracking branch 'origin/BDMS-470-Migrate-Stratigraphy' i…

ea1929e

…nto BDMS-470-Migrate-Stratigraphy

Formatting changes

9eab1c9

Copilot AI reviewed Jan 16, 2026

View reviewed changes

Comment thread transfers/stratigraphy_legacy.py Outdated

Comment thread transfers/stratigraphy_legacy.py

Comment thread tests/conftest.py

Update transfers/stratigraphy_legacy.py

76dcc29

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings January 16, 2026 18:58

Copilot AI reviewed Jan 16, 2026

View reviewed changes

chasetmartin reviewed Jan 16, 2026

View reviewed changes

Comment thread transfers/transfer.py

jirhiker added 2 commits January 16, 2026 16:08

Merge remote-tracking branch 'origin/BDMS-470-Migrate-Stratigraphy' i…

8845b30

…nto BDMS-470-Migrate-Stratigraphy

fix: remove dtype argument from read_csv call in transferer.py

b898e74

Copilot AI review requested due to automatic review settings January 16, 2026 23:11

Copilot AI reviewed Jan 16, 2026

View reviewed changes

jirhiker merged commit 38af393 into staging Jan 16, 2026
6 checks passed

TylerAdamMartinez deleted the BDMS-470-Migrate-Stratigraphy branch February 26, 2026 18:25

-        return str(value)
+        # Log when coercing non-string values to string to help catch data-quality issues.
+        logger.warning(
+            "Coercing non-string value to string in StratigraphyLegacyTransferer._string_value: "
+            "value=%r (type=%s)",
+            value,
+            type(value).__name__,
+        )
+        try:
+            coerced = str(value)
+        except Exception:
+            logger.exception(
+                "Failed to coerce value to string in StratigraphyLegacyTransferer._string_value: "
+                "value=%r (type=%s)",
+                value,
+                type(value).__name__,
+            )
+            return None
+        trimmed = coerced.strip()
+        return trimmed or None

Uh oh!

Conversation

jirhiker commented Jan 16, 2026

Why

How

Notes

Uh oh!

chatgpt-codex-connector Bot commented Jan 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants